System for operating and observing making use of mobile equipment

ABSTRACT

The invention relates to an operating and observation system, especially for an automation system, which is formed by at least one first operating and observation system having extensive functionality and a multiplicity of further operating and observation tools having limited functionality. The first operating and observation system having the extensive functionality is designed to be mobile, thereby doing away with mobility restrictions.

BACKGROUND OF THE INVENTION

[0001] The invention relates to a system for operating and observing.

[0002] Such a system and method are used, for example, in the field ofautomation technology, for production machinery and machine tools, indiagnostic/service support systems, and for complex components,equipment and systems such as, for example, vehicles and industrialmachinery and plants.

[0003] The contribution from Daude R. et al: “Head-Mounted Display alsfacharbeiterorientierte Unterstutzungskomponente anCNC-Werkzeugmaschinen”, [“Head-Mounted Display as a component to assistskilled operators of CNC machine tools”], Werkstattstechnik, D E,Springer Verlag, Berlin, Vol. 86, No. 5, May 1, 1996, pp. 248-252,XP000585192 ISSN: 0340-4544, describes the head-mounted display (HMD) asa component to assist the skilled operator with the steps of setting up,feeding and malfunction management in milling operations. The technicalintegration of the HMD into a modem NC control is explained, and theresults of a laboratory trial of the HMD are mentioned.

[0004] The object of the invention is to specify a system and a methodwhich, allows improved mobility in the operation and observation of, inparticular, an automatically controlled apparatus/plant/process.

[0005] This object is achieved by a system and by a method having thefeatures specified in claims 1 and 8, respectively. The invention isbased on the knowledge that in complex plants having spatiallydistributed individual components, but also in application situationswhere the mobility of the operator is important, the conventional use ofsystems installed relatively rigidly (e.g. monitors, operatingterminals) for the purpose of visualization and operation imposessignificant restrictions on the freedom of movement. As a rule, theoperator/skilled worker is forced by specific situations to leave hisnormal working situation in order to be able elsewhere to inform himselfof the current relevant process data or to carry out specificoperations. The use of mobile, portable equipment allows various workingsituations to be achieved in a more user-friendly manner.

[0006] The invention is based on the knowledge that, in complex plantshaving spatially distributed individual components, but also inapplication situations where the mobility of the operator is important,the conventional use of systems installed relatively rigidly (e.g.monitors, operating terminals) for the purpose of visualization andoperation imposes significant restrictions on the freedom of movement.As a rule, the operator/skilled worker is forced by specific situationsto leave his normal working situation in order to be able elsewhere toinform himself of the current relevant process data or to carry outspecific operations. The use of mobile, portable equipment allowsvarious working situations to be achieved in a more user-friendlymanner.

[0007] Depending on requirements, the deployment of a plurality ofmonitors these days either has them spatially distributed in the plantor concentrated together, for example in a control room. The deploymentof hitherto relatively rigidly installed systems for visualization andoperation in use is replaced as follows by novel appliances which areused in combination in an application-oriented manner: in situ, i.e.locally at the machine/plant component, a plurality of“minimum-capability devices”, specialized for specific applications, aredeployed.

[0008] The complete functionality for visualization and operationcovering the entire application is provided by means of high-performanceoperating units capable of mobile deployment. The essential advantage isto do away with the mobility restrictions to which the operator issubject. In addition, novel interaction techniques are employed . . .multimode (speech, gestures, . . . ). A situation of being tied to fixedterminals is replaced by mobile appliances, with the machine controlpanel being superceded/complemented by data goggles or at least beingused in combination with local terminals.

[0009] Advantageous refinements consist in the documentation data beingstatic and/or dynamic information data. Examples of such staticinformation include technical data from manuals, exploded views,maintenance instructions, etc. Examples of dynamic information includeprocess values such as temperature, pressure, signals, etc.

[0010] Rapid, situationally appropriate access to the documentation datais further assisted by the feature that the acquisition means include animage recording device, that the analyzing means are provided foranalyzing the real information in such a way that an operationalcontext, particularly an object of the documentation data, is determinedfrom the real information, and that the system includes visualizationmeans for visualizing the documentation data.

[0011] Rapid, situationally appropriate access to the documentation datais further assisted by the feature that the acquisition means include animage recording device, that the analyzing means are provided foranalyzing the real information in such a way that an operationalcontext, particularly an object of the documentation data, is determinedfrom the real information, and that the system includes visualizationmeans for visualizing the documentation data.

[0012] The deployment of augmented-reality techniques on the basis ofthe static and/or dynamic documentation data and/or process data can beoptimized for many applications by the acquisition means and/or thevisualization means being designed as data goggles.

[0013] The invention is described and explained below in more detailwith reference to the specific embodiments depicted in the Figures, inwhich:

[0014]FIG. 1 shows a block diagram of a first embodiment of anaugmented-reality system;

[0015]FIG. 2 shows a further block diagram of an embodiment of anaugmented-reality system; and

[0016]FIG. 3 shows a specific application for situationally appropriateaccess to expert knowledge and/or documentation data.

[0017]FIG. 1 shows a schematic depiction of an augmented-reality systemfor transmitting first information data from a first location O1 to aremote second location O2 of an expert for providing assistance to auser at the first location O1, for example in the event of servicingand/or repair, by the remote expert at the second location. The user,not explicitly shown in FIG. 1, is equipped with mobile equipment 4, 6.The mobile equipment 4, 6 includes data goggles 4 fitted with a videocamera 2 and a microphone 11. The data goggles are linked to a devicefor communication without the use of wires, for example a radiotransceiver 6, which can communicate with the automation system A1 . . .An via a radio interface 15. The automation system A1. . . An can belinked, via a data link 14, to an augmented-reality system 10,hereinafter also abbreviated to an AR system. The AR system includes aninformation module 1 b for storing or accessing information data, an ARbase module 8 and an AR application module 9. The AR system 10 can belinked to the Internet 5 via a data link 13, with optional access tofurther storage data and documentation data 1 a via an Internet link 12shown by way of example.

[0018] The user who is equipped with the data goggles 4 and the mobileradio transmitter 7 is able to move freely within the plant A1 . . . Anfor maintenance and service purposes. For example, if maintenance of, orrepair to, a particular subcomponent of plants A1 . . . An has to becarried out, appropriate access to the relevant documentation data 1 a,1 b is established with the aid of the camera 2 of the data goggles 4,optionally controlled by speech commands detected by the microphone 11.To do this, a data link with plant A1 . . . An or with an appropriateradio transmitter unit is set up via the radio interface 15, and thedata transmitted to the AR system 10. Within the AR system, the dataobtained from the user are analyzed in accordance with the situation,and information data 1 1 , 1 b are accessed automatically or in a mannercontrolled interactively by the user. The relevant documentation data 1a, 1 b obtained are transmitted via the data links 14, 15 to the radiotransmitter 6, with the overall result that an analysis is carried outon the basis of the operational situation detected, said analysisforming the basis for the selection of data from the available staticinformation. This results in a situationally appropriate,object-oriented or component-oriented selection of relevant knowledgefrom the most up-to-date data sources 1 a, 1 b. Information is displayedwith the aid of the visualization component used in each case, forexample a handheld PC or data goggles. Referred to AR-basedtechnologies. The operator in situ is therefore provided only with theinformation he needs. This information is always up-to-date. The servicetechnician therefore does not suffer from information overload from a“100-page manual”, for example.

[0019]FIG. 2 shows a further specific application of a documentationprocessing system for service and maintenance. The system consists of anaugmented-reality system 10 which comprises an information module 1 bfor storing information data, an AR base system 8 and an AR applicationmodule 9. The AR system 10 can be linked to the Internet 5 viaconnecting lines 13, 18. Thence a link is possible, via an exemplarydata link 12, to a remote PC 16 with a remote expert 22. Linkage betweenthe individual modules of the AR system 10 is effected via links 19, 20,21. The user communication between a user 7 and the AR system iseffected via interfaces 8, 23. To this end, the AR system can be linkedto a transceiver which enables bidirectional data communication betweenthe AR system 10 and the user 7 via data goggles 4, either directly viathe interface 8 or via a radio transceiver 17, located in the vicinityof the user 7, via an interface 23. The link 23 can be implemented via aseparate data link or via the mains as a “power-line” modem. As well asa display device disposed in the vicinity of the eye pieces, the datagoggles 4 comprise an image recording device 2 in the form of a cameraand a microphone 11. With the aid of the data goggles 4, the user 7 canmove round the plants A1 . . . An and carry out service or maintenanceactivities.

[0020] With the aid of the data goggles 4 and the corresponding radiotransceivers, e.g. the radio transceiver 17 worn by personnel directlyon the body, it is possible to achieve preventive functionality: theinitial step is the detection of the respective operational situation,for example by the camera 2 or via location by the personnel 7. On thebasis of the operational situation detected, a selection of data fromthe plant A1 . . . An undergoing maintenance is made in the AR system.The fundamental advantage of the system depicted in FIG. 3 is that thissystem assists the cooperation of the individual single functionalitiesin an application-relevant manner: i.e. a concrete operational situationis first detected automatically, and this operational situation is thenanalyzed, the aspects relevant at that point being determinedautomatically from the most up-to-date available static information inconjunction with the dynamic data acquired instantaneously. As a result,for example, assembly suggestions are correlated with current processdata. As a result, personnel 7 are provided with a situationallyappropriate display of the relevant information, for example by asuperposed visualization of the respective data in such a way that thereal operational situation in the field of view of the personnel isexpanded by the information acquired. As a result, personnel 7 are veryrapidly put in the position of being able to act, thereby ensuring therequisite machine operating times. Assistance to the maintenancetechnician 7 in situ can also be provided via the remote expert 22 andthe knowledge 16 available at the location of the remote expert 22.

[0021]FIG. 3 shows a specific application of situationally appropriateaccess to documentation data. FIG. 3 shows a first monitor region B1which shows a plant component. Shown in the right-hand monitor region B2is a user 7 who, for example, is looking at an individual plantcomponent. The user 7 is equipped with data goggles 4 which comprise acamera 2 as acquisition means. Additionally disposed on the data goggles4 are a microphone 11 and a loudspeaker 16. The left-hand monitor regionB1 shows a view of conduits which can be viewed with the data gogglesshown in window B2. Marked in the left-hand monitor region B1 are twopoints P1, P2 which each represent two image details viewed with the aidof the data goggles 4. After the first point P1 has been viewed, i.e.after the conduit disposed at or near point P1 has been viewed,additional information is visualized for the user 7 in the data goggles4. This additional information 11 consists of documentation data which,regarding the first point P1, include operational instructions for thispipe section and, regarding point P2, comprise the installationinstruction to be implemented in a second step. The installationinstruction in this case consists of the user 7 being informed of thetorque and the sense of rotation of the screwed joint of point P2 viavisualization of the additional data 112. The user 7 is therefore veryquickly provided with situationally appropriate instructions for theobject being-viewed. If an intelligent tool is used which is able todetect the torque applied at any given moment, it is also possible forthe user to be told, on the basis of the current torque, to increase orreduce the torque as required.

[0022] Below, background information is provided to the field ofapplication of the invention: this involves an application-orientedrequirement analysis and development of AR-based systems to supportoperational processes being developed, production and service of complexengineering products and plants in fabrication and process technology,and for service support systems as with motor vehicles, or formaintaining any industrial equipment.

[0023] Augmented reality, AR in brief, is a novel type of man-machineinteraction of major potential for supporting industrial operationalprocesses. With this technology, the field of view of the observer isenriched with computer-generated virtual objects, which means thatintuitive use can be made of product or process information. In additionto the extremely simple interaction, the deployment of portablecomputers opens up AR application fields involving high mobilityrequirements, for example if process, measured or simulation data arelinked to the real object.

[0024] The situation of German industry is characterized by increasingcustomer requirements in terms of individuality and quality of productsand by the development processes taking substantially less time.Especially in developing, producing and servicing complex industrialproducts and plants it is possible, by means of innovative solutions toman-machine interaction, both to achieve progress in efficiency andproductivity and to design the work so as to enhance competence andtraining, by the user's need for knowledge and information beingsupported in a situationally appropriate manner on the basis of dataavailable in any case.

[0025] Augmented reality is a technology with numerous innovative fieldsof application:

[0026] In development, for example, a “mixed mock-up” approach based ona mixed-virtual environment can result in a distinct acceleration of theearly phases of development. Compared with immersive, i.e. immersing“virtual reality” (VR) solutions, the user is at a substantial advantagein that the haptic properties can be depicted faithfully with the aid ofa real model, whereas aspects of visual perception, e.g. for displayvariants, can be manipulated in a virtual manner. In addition, there isa major potential for user-oriented validation of computer-assistedmodels, e.g. for component verification or in crash tests.

[0027] In flexible production it is possible, inter alia, toconsiderably facilitate the process of setting up machinery forqualified skilled operators by displaying, e.g. via mobile ARcomponents, mixed-virtual clamping situations directly in the field ofview. Fabrication planning and fabrication control appropriate to theskilled worker in the workshop is facilitated if information regardingthe respective order status is perceived directly in situ in connectionwith the corresponding products. This also applies to fitting, with theoption of presenting the individual procedural steps to the fitter in amixed-virtual manner even in the training phase. In this connection itis possible, e.g. by comparing real fitting procedures with results ofsimulations, to achieve comprehensive optimizations which both improvethe quality of operation scheduling and simplify and accelerate thefitting process in the critical start-up phase.

[0028] Finally, regarding service, conventional technologies are by nowbarely adequate for supporting and documenting the complex diagnosticand repair procedures. Since, however, these processes in many fieldsare in any case planned on the basis of digital data, AR technologiesprovide the option of adopting the information sources for maintenancepurposes and of explaining the dismantling process to an engineer, e.g.in the data goggles, via the superposition with real objects. Regardingcooperative operation, the AR-assisted “remote eye” permits adistributed problem solution by virtue of a remote expert communicatingacross global distances with the member of staff in situ. This case isparticularly relevant for the predominantly medium-sized machine toolmanufacturers. Because of globalization, they are forced to set upproduction sites for their customers worldwide. Neither, however, is thepresence of subsidiaries in all the important markets achievable oneconomic grounds, nor is it possible to dispense with the profoundknowledge of experienced service staff of the parent company withrespect to the increasingly more complex plants.

[0029] The special feature of man-machine interaction in augmentedreality is a very simple and intuitive communication with the computer,supplemented, for exam-ple,by multimode interaction techniques such asspeech processing or gesture recognition. The use of portable computerunits in addition enables entirely novel mobile utilization scenarios,with the option of requesting the specific data at any time via awireless network. Novel visualization techniques permit directannotation, e.g. of measured data or simulation data, to the real objector into the real environment. In conjunction with distributedapplications, a number of users are able to operate in a realenvironment with the aid of a shared database (shared augmentedenvironments) or to cooperate with AR support in different environments.

[0030] Augmented reality has been the subject of intense research onlyin the last few years. Consequently, only a few applications exist,either on the national or the international level, usually in the formof scientific prototypes in research establishments.

[0031] U.S.A.: As with many novel technologies, the potential uses ofaugmented reality were first tapped in North America. Examples includecockpit design or maintenance of mechatronic equipment. The aircraftmanufacturer Boeing has already carried out initial field trials usingAR technology in the assembly field. The upshot is that in this hi-techarea too the U.S.A. occupy a key position, potentially making themtechnological leaders.

[0032] Japan: Various AR developments are being pushed in Japan, e.g.for mixed-virtual building design, telepresence or “cyber-shopping”. Thenucleus is formed by the Mixed Reality Systems Laboratory founded in1997, which is supported jointly as a center of competence by scienceand by commerce and industry. Particular stimuli in the consumer goodsfield are likely in the future from the Japanese home electronicsindustry.

[0033] Europe: So far, only very few research groups have been active inEurope in the AR field. One group at the University of Vienna is workingon approaches to mixed-real visualization. The IGD group, as part of theACTS project CICC, which has now come to an end, has developed initialapplications for the building industry and a scientific prototype forstaff training in car manufacturing.

[0034] The invention in particular should be seen in the specificcontext of the fields of application “production machinery and machinetools” (NC-controlled, automation-technology processes) and“diagnostics/service support systems for complex engineeringcomponents/equipment/systems” (e.g. vehicles, but also industrialmachinery and plants).

[0035] In complex plants having spatially distributed individualcomponents, but also in application situations where the mobility of theoperator is important, the conventional use of systems installedrelatively rigidly (e.g. monitors, operating terminals) for the purposeof visualization and operation imposes significant restrictions on thefreedom of movement.

[0036] As a rule, the operator/skilled worker is forced by specificsituations to leave his normal working situation in order to be ableelsewhere to inform himself of the current relevant process data or tocarry out specific operations. The use of mobile, portable equipmentallows various working situations to be achieved in a more user-friendlymanner.

[0037] Depending on requirements, the deployment of a plurality ofmonitors these days either has them spatially distributed in the plantor concentrated together, for example in a control room.

[0038] The deployment of hitherto relatively rigidly installed systemsfor visualization and operation in use is replaced as follows by novelappliances which are used in combination in an application-orientedmanner:

[0039] in situ, i.e. locally at the machine/plant component, a pluralityof “minimum-capability devices”, specialized for specific applications,are deployed.

[0040] The complete functionality for visualization and operationcovering the entire application is provided by means of high-performanceoperating units capable of mobile deployment. The essential advantage isto do away with the mobility restrictions to which the operator issubject. In addition, novel interaction techniques are employed . . .multimode (speech, gestures, . . . ).

[0041] Instead of being tied to fixed terminals, mobile equipment isused.

[0042] Supercession/complementation of machine control panel by datagoggles.

[0043] Deployment in combination with local terminals.

[0044] In summary, the invention therefore relates to an operating andobservation system, especially for an automation system, which is formedby at least one first operating and observation system having extensivefunctionality and a multiplicity of further operating and observationtools having limited functionality. The first operating and observationsystem having the extensive functionality is designed to be mobile,thereby doing away with mobility restrictions.

We claim:
 1. A system for operating and observing, comprising at leastone first operating and observation system having extensivefunctionality and comprising a multiplicity of further operating andobservation tools having limited functionality, wherein the firstoperating and observation system having the extensive functionality isdesigned to be mobile and, to achieve full performance, is designed tobe coupled with the further operating and observation tools.
 2. Thesystem according to claim 1, wherein the mobile operating andobservation tools can be driven by multimode interaction techniques. 3.The system according to claim 1, wherein the system includes acquisitionmeans comprising an image recording device for acquiring firstinformation data and visualization means for visualizing secondinformation data.
 4. The system according to claim 3, wherein theacquisition means are user-controlled.
 5. The system according to claim3, wherein the acquisition means are designed as speech-controlledacquisition means.
 6. The system according to claim 3, wherein theacquisition means are designed as acquisition means controlled bycontrol data.
 7. The system according to claim 3, wherein thevisualization means are designed as display devices arranged in the areaof eye pieces of a pair of data goggles, in that the acquisition meansare designed as an image recording device arranged on the data goggles,and in that a microphone is provided on the data goggles in order todetect speech commands.
 8. A method of operating and observing, by meansof augmented-reality techniques, an automatically controlled apparatus,wherein the automatically controlled apparatus can be controlled via afirst operating and observation system having extensive functionality incombination with a multiplicity of further operating and observationtools having limited functionality, wherein the first operating andobservation system having the extensive functionality is designed to bemobile and, to achieve full performance, is designed to be coupled withthe further operating and observation tools.
 9. The method according toclaim 8, wherein the mobile operating and observation tools can bedriven by multimode interaction techniques.
 10. The method according toclaim 8, wherein first information data are acquired by means ofacquisition means comprising an image recording device, and in thatsecond information data are visualized to the user by means ofvisualizing means.
 11. The method according to claim 8, wherein theacquisition means are user-controlled.
 12. The method according to claim10, wherein the acquisition means are designed as speech-controlledacquisition means.
 13. The method according to claim 10, wherein theacquisition means are designed as acquisition means controlled bycontrol data.
 14. The method according to claim 10, wherein thevisualization means are designed as display devices arranged in the areaof eye pieces of a pair of data goggles, in that the acquisition meansare designed as an image recording device arranged on the data goggles,and in that a microphone is provided on the data goggles in order todetect speech commands.